Repository navigation
fix(quality): keep the gate verdict if the summary write fails - #237
Merged
Merged
Conversation
A GITHUB_STEP_SUMMARY that cannot be written no longer ends the gate under set -e before the verdict and survivor listing; it warns on stderr. The gate test rule now inherits GITHUB_STEP_SUMMARY, which Bazel's test sandbox otherwise drops, and the stale timeout figures leave the cap comment. Generated-by: LLM effort=medium;harness=claude-code:2.1.318;model=anthropic:claude-sonnet-5-5;session=sha-256-64:f1227f050617f845;surface=sdk-cli
Generated-by: LLM effort=medium;harness=claude-code:2.1.318;model=anthropic:claude-sonnet-5-5;session=sha-256-64:f1227f050617f845;surface=sdk-cli
The summary warning's own write to stderr could end the gate before its verdict when stderr was closed. Ignore that failure, require the FAIL line in the failing summary test, say no mutant failed when none was checked, and reword the header. Generated-by: LLM effort=medium;harness=claude-code:2.1.318;model=anthropic:claude-sonnet-5-5;session=sha-256-64:06dc52093e5d8aba;surface=sdk-cli
tbhb-agent
added a commit
that referenced
this pull request
Nov 14, 2026
Why The gate read each report with `grep ... || true`. That treats grep status 2 (an unreadable file or a broken pattern) like status 1 (no match), so a failed read looked like a clean report. Under fault injection, six fixtures that must fail exited 0. Review of #237 found it, and it predates that PR. Behavior change Every verdict read goes through one helper. Status 0 and 1 pass as before. Any higher status prints the report and the step on stderr and exits 1. The helper serves the value reads, the mutant and count reads, the scope read, the unmapped count, and the timeout listing. A failure in any stage of the unknown-status pipeline fails the gate. Reports that parse cleanly get the same verdict and exit status as before. The comment before the final message now separates a diff that selected nothing from an all-skipped report, which also checks nothing in full scope. Decisions - The timeout listing fails closed too. A listing that can't run would hide the timeouts it exists to show. - The summary-write guard from #237 keeps its form, because it protects a file write and the gate verdict comes from the report. - One helper serves every site. A grep added later has one obvious way to handle status. - Tests put a shim ahead of the real tool on `PATH` that returns an exact exit status. File permissions are unchanged. Risk Low for well-formed reports. A gate on a host whose grep fails for an unrelated reason now fails where it used to pass, and that's the intent. Verification Six status-2 fixtures exited 0 on the previous main, and all exit 1 now. Thirteen fault tests inject status 2 at the grep sites and at each stage of the unknown-status pipeline. A shim prints valid output before it fails, and one test uses status 3. Putting back `|| true` on a count read, on the pipeline or on the scope read fails the matching test with `the gate exited 0, not 1`. The faults come from shims. The tests never read a real unreadable file. A different build host ran the builder checks, because the usual one was unreachable. Generated-by: LLM effort=medium;harness=codex:0.164.0;model=openai:gpt-6.1-sol;session=sha-256-64:44399538b20b63d8;surface=exec Assisted-by: LLM effort=medium;harness=claude-code:2.1.322;model=anthropic:claude-sonnet-5-5;session=sha-256-64:debb814921f029ad;surface=cli
tbhb-agent
added a commit
that referenced
this pull request
Nov 14, 2026
Why A pull-request mutation lane needs to report surviving mutants without blocking the merge. The gate fails on any survivor today, so a lane can't report one. Nightly lanes must keep blocking. Behavior change `mutation_gate_test` takes `advisory`, which defaults to False. With it set, lived and not-covered mutants no longer fail the gate. It lists each survivor and prints `mutation_gate: advisory, N survivors (not failing)` on stdout and in the step summary. Both runner macros, `mutmut_test` and `gomutants_test`, forward it. Errored mutants, interrupted runs, bad accounting, malformed reports, timeouts over the cap, unmapped mutants, an empty full run and a diff report under full scope still fail. Decisions - Advisory covers survivors only. A second advisory case needs its own PR. - The docstring says blocking nightly runs leave `advisory` False. - The advisory line reuses the guarded summary append from #237. A failed summary write gives a stderr warning and exit 0. - The survivor listing goes through the helper from #239. A failed read fails the gate. Risk Low while `advisory` is False. A lane that turns it on sees survivors as a passing check. Verification Each non-survivor failure still exits 1 in advisory mode, and nine proofs cover them. The same survivor report exits 1 without advisory. Forcing advisory on fails the proof that survivors fail without it. Ignoring advisory fails the survivor test. Letting advisory wave through errored mutants fails the errored proof. The build ran on a different build host because the usual one was unreachable. Remote CI, other operating systems and runner integration with advisory on are unverified. Generated-by: LLM effort=medium;harness=codex:0.167.2;model=openai:gpt-6.1-sol;session=sha-256-64:d27b03000ac671ee;surface=exec Assisted-by: LLM effort=medium;harness=claude-code:2.1.322;model=anthropic:claude-sonnet-5-5;session=sha-256-64:debb814921f029ad;surface=cli
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
mutation_gate_testappends its timeout line to$GITHUB_STEP_SUMMARY. After #233, a summary path that failed to write ended the gate before its verdict. Bazel's test sandbox also dropped the variable, so the summary got nothing.Behavior change
A failed write to the summary now prints
mutation_gate: could not write to GITHUB_STEP_SUMMARYon stderr. The gate still prints its timeout line and survivor listing, and it exits with the verdict the report deserves.mutation_gate_testnow inheritsGITHUB_STEP_SUMMARYfrom the environment. The timeout line lands in the job summary when CI sets the variable. The change also drops a comment that quoted one run's timeout figures.Decisions
RunEnvironmentInfo(inherited_environment = ...).env_inheritbelongs to native test rules, and this rule is custom.Risk
Low. The change touches the append and the environment list. Bazel keys the test's cache on the inherited value, so a new summary path reruns the gate test.
Verification
A scratch file as
GITHUB_STEP_SUMMARY, one gate test, no--test_env: before the change the file was empty, and afterward it heldmutation_gate: 5 timed out of 100 (5.0%), counted as killed.With the stderr guard removed, both new tests fail. The passing-report test exits 1 instead of 0, and the failing-report test doesn't find the stderr note.
The tests cover a missing directory only, with no read-only file. CI doesn't show whether a gate inside a real job receives the variable.
Opened-by: LLM effort=medium;harness=claude-code:2.1.322;model=anthropic:claude-sonnet-5-5;session=sha-256-64:debb814921f029ad;surface=cli
Generated-by: LLM effort=medium;harness=claude-code:2.1.318;model=anthropic:claude-sonnet-5-5;session=sha-256-64:f1227f050617f845;surface=sdk-cli
Assisted-by: LLM effort=medium;harness=claude-code:2.1.322;model=anthropic:claude-sonnet-5-5;session=sha-256-64:debb814921f029ad;surface=cli